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ABSTRACT 

In an examination of the implementation oX^ 
educational change, a discussion is presented on the purpose, 
meaning, and context of change. ^Consideration, is given to the value 
of change, its benefits and f e.asibility, and the capa'city for 
implementation.. In the, seco^nd section of the paper, 14 factors 
related to the implementation of change in schools are summarized, 
and corresponding research on these factors £s cited. In discussing 
the characteristics of the change itself, the factors ot need, 
clacity, complexity, quality, and practicality are examined. <Six 
factors influencing the charac;teristics of change at the school 
district level are discussed: (1) the district's history of 
innoT^ative attempts; (2) the , adoption process; (3) district x 
administrative supjport; (4) staff /development and participation; (v5) 
time-liiie and information systems; and (6) commur^^ity characteristics. 
In discussing the school level factors relating to change, the rple' 
of the principal*, teacher-^te'acher relationships, and teacher 
characteristics and orientations are singled out as factors for 
consideration. The last set of factors are considered under the label 
of assistance external to the school district. Six types outcomes' 
of change are identified: (1) degree of organizational change; (2) 
scope of implementation; (3) incorporation of the product; "^U) 
incorporation of a problem-solving process; (5) problem resolution; 
and' (6) personal impacts. The final section of the paperi addresses 
implications, unresolved issues, and tHe question of deriving 
practical l^ssonsf or integrating the theory and practice of \ 
educational change. (JD) 
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Over. a decade ago, a trilogy- of studies on implementation - 
GrossM1971), Sarason (1971) and Smith and Keith (1971) - all well 
done and provocative, launched a new wave of interest and researfch 
/on the problem of educational change. They were studies of the 
^failure of educational change* in practice. As iJisightful a? these 
studies were thev were only of. incho^ value. We found out during 
the 1970 \s that there were many different ways to fail» ^^nd being 
able to^ explain failure was not of direct help in being able to 
understand, let alone influence success. * In 1982 I believe we can 
"honestly say that 'Wd can understand success, and ev«n help bring it 
about under certain conditions (the latter of course being a 
signifit:ant qualifier) , , ' • ' * 

' - # 

I propose to ej>amine the question of .iinplernentation under three 
headings: First, a consideration of the purpose, meaning ^ context of change; 
the second and core aspect summarizes findings/from ^implementation 
research per _ so; »the final section of the paper addresses implicationn, 
unresolved issues', and the quest ion o-f deriving ^practical lessons 
for integrating the theory and practice of educational change. 
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I T!IR PURPOSE, MEANING AND CONTEXT OF CHANGE 

I « * 

» f 

f 

' < f • 

' ^ ' 1 

There arp three fundamental points to be made at the outset 
« * 

pertaining respectively to puiposc, meaning and context. -First, ^ 

in theory, the purpose of educationvTl change is to help schools 

accomplish their goals more effectively by replacing some programs ^ 

or practices v/ith better ones. Hov/ever, in actual practice it is * , 

not at, all clf.ar what is better, and for Whoiiu VJe cannot assume 

tr.:it ^ny given change i3 worthwhile and Vhould be implemented. In 

examining this matter wc should keep in mind tv;o, central questions: 

v;ho benefits from the chancje (the .values question) ? And', *How sound 

6r feasible is the idea and the program ot approach (the capacity for 

implencntatJOii question)?. Both arc complex and difficult questions 

to- answer. To highlight the problem consider figure 1>. 

/ Figure 1 Types of Implementation Outcomes of 

Adopted changes ' ' ^ ' 

Actual Implementation 



Valjje and technical 
\quality of the change 
1 ' 



(From Fullan, 1982:14) 
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"Actual implementation" refcirs to whether or not there has 
been real change in practice. "Value and technical quality" collapse 
the two factors related to who benefits and whether or not the program 



has been technically well deyeloped.* 

Type I in figure 1 represents. what we are presumably striving 
for: actual implementation of .(ajguality program which we value, 
^ Type II reflects a planning problem in that a worthwhile prpgram 
for whatever^ reasons is not v^rkihg. We do not often conceive of 

^ Ti'pcs III .and IV. In the ^ former situation a change which' is not 
woll^ developed and/or valuod is being put into' practice. In short 
a bad chance is b.eing introduced. Type IV, interestingly, is a form 

, of suc'ccss i'n thcit a poorlv valuod or poorly developed change is bi. in 
rejected in practice. In any case, the poi^t^is that whether a 
particular po^,6ntial change program or direction is desirable is 
ah open question to be dotormi ne^d 4r/^ih*,' variety of people most 
affected. 

The second main point is that change happens to individuals. 
Every change has two components : -^^-implicit or explicit "theory 
of education*' (what the chance. is) and an implicit or explicat 
•;th^ory of change" (tl^he process being followed to implement it) • 
Individuals must fincJ meaning in both aspects, if change is to 
.s^ucceed. No* matter who decides on thfe direction of change - teachers 
'administrators, external developers or policy- makers the* meaning of 
change in terms of acceptance, rejection, and modification jr^ustj^f 
be confronted and worked through. 'I leave 'open the question of 
wl^ether change, is initiated internal or -external to a school or ^'a 



The two"facJ:ors of who benefits and technical quality must be 
separated in any furWjer analysis, ^and are combined here for > 
the sake of s;Li]nprlicity^ in' making tHe main point. ^ 



school distric*t. The fact is that changes come from both internal 
aj^ 'External sources r an.* must- be a'Ssessed.4:>n!~thelr-45artjL^cuaar ^ 
merits from each ind|.yidual • s or group's perspective. Neither 
inter-nal or external change is intrAnsiqally good or bad. 

The third basic issue is* that educational change is context 
bound. The history, personalities, and socio-political climate 
*vnthin each setting constitute major determinants of change outcoracs 
There are a number of common fcactors related to success and failure, 
but in any individual situation both comiror: and unique knov^iedge 
is necessary (seq. Lindblom and Cohen', 1979) . • . 

' Th§ jncr.'^ago in the rest of irhis paper is tKofold: (1) don't 
launch ;headlona into somethina without fir^t checking out many 
factors kn^wn to affect the chances for success, and (2) don't 
be afraid to try pomethanq new armed with -the knowled.ge' carrered 
from, the research and practice of educational change over the past 
decade*. Let us now turj. to what research on implementation has tp 
tell- us,. ' * ; 
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I] implementatijOn. research 

. ' • • ^ 

Implementation, consists of the process of putting into 
prajzftice (or. failing to doso) an idea, program* pr • set of 
activities hew to the people attempting or expected, to^change. 
It may be externally imposed or voluntarily sought, explicitly ^ ^ 
defined in detail in advance, or developed and adaptfed 
. incrementally throu*gh use, designed to be used uniformly, or 
deliberately planned so that users can specify it accor/2ing to 
their perceptions of the needs of the situation . We can divide 
' implementation into 'two broad components: factors affecting 

implementation, and implemen.tation outcomes — ' the latter being 
^ referred to increasingly under Jthe -rubric of school -improvement. 
I wiil return to and elaborate, on the concept of implementation 
outcomes in the latter part of this section, but let me suggest 
.for starters that one ma^or type of outcome is "change in practice 
/ I tend '.to. view change in practice .as being multidimensional 

consisting of at. least three, related aspects (1) changes in the 
use of new resources in the classroom (curriculum materials, new 
'technology etc.); (2) change in classroom behavior (e.g., new 
steills and teaching behaviors in relation to the change);, and (3J 
-alterations 'in beliefs., theories or assumptions associated with 
the change. Implementation* is so difficult because.it involves 



Vor the sake of simplicity I will focus on changes in 
clas.srooin practice'* ^ * 
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potential changes,, in what people do (using new resources and- 
new teaching approaches )' and in what they think' ..(altering one's . 
beiiefs and -educational theories).* Most directly then, the 
question of implementation is one which «asks : which f actors^f f ect 
the likelihood that there will be ch'anges in what, people do and ^ 
think in relation to their educational practices. In the rest of 
this section I will first' take up .the topic of what we know a^bouf . 
factors af fectAnct^impiementation, and second, .conclude with a 
discussion of impiemcntation outcomes. ^ . 

Factors Affectina' Iir.pleirentati''on 

Although the process of implementation is complex in any 

given situation and there are many ^dif f erent specif i-c v'^rLfples, 

, • . »• . 

permutations and combinations, it is possible to summarize in. a 

. ••' ./ ■ . ■ 

relatively comprehensive way the main factors. This involves^ some 

abstraction, and there are .different ways of organizing thfe \^ ■ 

clusters of variables involved. However, ipuch of the resGarcK\ 

over the past"t4n years . (and increasingly so.) is^ consistent in \ 

identifying a/number of factors related to change in practice. \ 

Table 1 contains a list of these factors 'as 1 Have organized them " 

(for a slightly different but congruent ;summary see Herman, 1981) . ' 

In effect; table 1 suggests that change in practice is a function 



*ln a more Complete treatment the topic, I define ^he problem 
of educational changes as the problem of meaning . - of , the difficulties 
and conditions under which individuals can come to unaerstand the 
why and how of doing something hew (See .Fullan , 1982) . 



F^ACTORS AFFECTING IMPLEMENTATION 




CHJ^RACTERISTICS OF THE CHANGE 

1. Need , ' . ' 

2. Clarity , 
* 3. Comp3exity 

4. .Quality and practicality 

CHARJvC-rliRISTICS AT THE SCHOOL. DISTRICT LEVEL ' 

5. The district's history of innovat-ive attempt 

6. The adoption process _ . ^ 

7. District admiijistrative support* ^ ^ 

8. Staff development and participat^-oh 

9. T±me-llne and Information Systems 

10. ComiTiunity characteristics- 

S^OOL LEVEL FACTORS 

11. The role of the principal- 

12. Teacher-teacher rel-ationships ^ ' . 

13. teacher characteristics and orientations 

% 

EXTERNAX FACTORS 

14. External assistance 
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of four types of characteristics pertaining to:. A. The 'Change' 

being attempted, B» .The Scho^ol District, C- ^^The ochool Levei, and ^ 

p. External Assistance. It will only;, be possible in this** paper 
« » 

to suiranarize briefly' the nature *of the 14 fa.ctors and corf esx>onding 
i^ese^rch.* 

/ • . ' ■ > . 

/ 

A, C haracteristics of ^tho Chance ' * \ 

1. Need \ . . 

Many innova-tions are attempted without a car*eful 
« , examination of whether they address particular priority 
• # . needs. Teachers fbr 'example; frequently do not see the 
need for a change that is being advocated. Several 
large scale Studies'' confirm the importance of relating 
need to decisions about innovaljions or change directions 
In the Experimental Schools (ES,)^ project Rosenblum and 
Louis {.1979:1*^) found -that^ ''the degree to which there was 
a formal recognition within the school system of* unmet ^ 
needs** was one of the four ''readiness factors" which was 
associated with subsequent implementation. The Rand 
Change Agent Study (FPSEC) identified problem solving 
decision-making (f.e. , idejitif ication of a need linked 
to. selection o£ a^program) as strongly , related to successful 



\ 



. The, following discussion is somewhat cryptic due to limitations of 
space.-. Each o^f th.e factors should' be discussed at more length 
because there, are many qualifiers and nuances involved. 
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implementation {Berman and McLaughlin, 1978). The RDU 
project reports that perceived relevance of products is 
correlated significantly .with extent of implementation ^ 
(Louis and 'Rosenblum,, 198.1)^ Other studies have 
discovered that implementation is moire effective when 
^ relatively fpcussbd or specific needs are identified. 

.(s.g.r Emrick and Peterson, 1978, Louis and Siober , 1979) . . 
Of coiirsey the jSroblcm is that n^ed may ^.not be evident at' 
the bcginriing ofljthe change process, but may b'^ created as 
people get more familiar with a particular ^change (seQ , 
•especially the case studies in Crandall et al's 
forthcoming research on Dissemination Efforts Supporting, 

"* ' ' ' • . 

School Improvement (DESSI) .-^ 
2. Clarity * * • : ' ^ ^ 

Clarity, (about goals -and means) is 'a perennial problem 
in the change process ^ - Even -when there is a potential need^ 
as when teachers want to improve some area of the curriculum, 
t:he 'change may be- not at all clear about what the'y should 
do differently.* Gross et al (1971) founi that the ' 
• majority of teachers .were unable to identify j^he essential 
^features of the innovation they .were.^using. ^Problems 
. related to clarity have been found In vfrtuaTlT^evi^ study 



\ 



The change,^ for instance, may^ not be sufficiently well developed 
to be "implementable". \ - 
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t - of ~sl-gni-f-i-Gan.t^xh.ange_Je^. , A'oki,-, et a-1, 1977^p-§karters 

and P-ellegrin, .1973, Miles , <d978 T^i^lb^Tj—Weath^rle^^^ 



1979), In short, lack- of clarity , /diffuse goals and ♦ 
unspecif ied means of . impl^jnentati^^n - r.epresents a ma^or 
problem, at "the implementation stage at which time teachers 
' and others find that the change is simply not very^ clear 
as to v;hat it means in pr.^^i:ce-j-:^_^^ ^ ^ v* . ' . 

Clarity, of course, cannot be delivered on a platter. 
It is accomplished or not depending on the process .* 
\Nor is greater' clarity an end in itself. Very simple 
•and sigYiificant changes can be v'^ry clear, v'hile mere 
difficult and worthwhile ones, may not be amenable to 
easy clarification.. This brings me direct]^^^^toJ:he--th-ixld. 
relat.e.d^factor r- complexity.. 



3. Complexity . ••. • 

, \ Complexity re-fors to the" difficulty and extent of 



' change^wlrich might be involved for; individuals engaged in 
i^jplementation. " The actual ambuntrdfepSnds on the sta^rting 
point for any given individual ok grpup, but the main idea 
is that any change can be examined ijp regard' to the . 
difficulty, skill required, an,d exte'rvt of alterations in - 
beliefs, teaching strategies, and iise^of material^trr 



*The r,elationship to other factors in categories B and C is pbvious 
To take one example^, sta-ff development activities if approppriBte, _ 
contribute "to greater clarity-, . • \< ^ 



cnanges, such as open, education {Bussis et al, 1976) ^ * : ^ 
systematic direct instruction fGersten et al, 1981), ' 
' . , inquiry! oriented social studies (Aoki et al, 1977) > special 
c - * ecJucatic^n ^Weatheirley, 1979), parent involvement- (Fax^tihi^ . ; 
' 1980) , require a sophisticated array of activlti os'i 



diagnosis^ teaching, strateaaes and philosophical 

tinderstariding in otder to achieve , effective impLementation . 
^ ' " ' r • : ' > 

VJJiile complexity create? problems for implenentation r 

' it may'^result in greater change because more is being 

. .attempted. Herman and McLaughlin (1977 :,g8) found the • 

interesting combination that "ambitious projects were less 

successful in absdXute terms of the percent of project 

, goals achieved, but th^^"""l;ypic^ stimulated more teacher 

' change than projects attempting less^v...^^ose changes which 

vdid occur Were more thorough as a result of the^^xtra 

. effort which the project required- or inspired. Ar^Ber^nan 

^ (1979) stated it elsewhere, "li-ttle venjtured,. hothing ^ 

gained"\ In the DESSI^study / Crandall . et al ^(forthcoming) 

, discovered that it was necessary to separate teachers 

' attempting "a Ifnajbr, change from those attempting a minor 

change before ^discernible patterns of explanation could be 

found.* It is the case that s.imple changes are easier tp 



Degree of change was defined in terms of the individual statting 
points of teachers. Thus using the same innovation some teachers 
faced major change^ and others. minor 'change depending on their 
starting points. / " ^ 
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' / \ to carry out, but they, may .not make lormch of a difference. 
Relatively complex changes prcmisfe to accomplish .more* 
>'hethdr^vtHey do .or not depends on the combinajb^on pf 
factors • discussed ' in this' section . 

I. Quality and -Practicality ^ 

— TRe l a ^ 't'^f'^ctior^^s^r)^^ t he a alL:ir^ 

* * of chanae concerns the quaTity and'^practicality ^of 

Material S; products (or whatever form the change ^>.come"S^ 
.■±h) . The Icircc scale evaluation of innovations adopted 
■ through the Nntional Diffusion Network (NDN) s'peaks to 
the issue: <,;'well articulated adoption materials , which. . . 
are complete, well organized, comprelionsive and deta iled" , 
and address "how to" concerns are more effective at the 

1 ' ' ' 

. implementation stage; at earlier phases,, concise, overview 

materials are better (Emrick. et al, 1977, Emrick and 

Peterson, 1978:73). The^RDU (Louis and Rosenbl-um, 1581) 

and/DESSI (Crandall et al forthcoming) studies also confirm 

/ * * ' ^ 4 

*tha|t product quality, is an important ^Factor in relation 

\ to jdhange in practice^'* * . ' . 

• \ The history and role of product pr program quality is 
not\straightforward. ; The rush for innovation in the earlier 
peribd of educational change (the 1960's) probably resulted 

*I do net examine questions of measurement in this paper. Quality 
can be measured in terms of "perceived quality" by users, or - . 

■throufl some .system of external validation (e.g., Joint Dissemination 
Review Panel). " 
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tit insu:^f icieirtx^ develpped'/or^^or quality changes- 
; (however nfi.ede4_the £ariicul|ir...chang.e..di might 



< 



, have^^been)\ There is al^o the more subtle, problem to , 
vi7hicti I have referred elsewhere as "the (3ilemma.XLf 
explicitness" (Fullan and Pomfret, 197 7:368-9). To 
make innovations highly explicit at the de.velopr.ent stage 
may mean that they are inappropriate for the variety of 
settings faced by teacher.s. For many proiplems, the 
situational knowledge of teachers is essential to decidin'g 
on the specific form of change (see Connelly and Elbaz, 
1980, Huberman, 19S0, Roberts., 1980). On t.he other hand, 
to leave innovations general, results in great confusion 

. - ^ ^ ^ 

about v/hat to do in practice. One of the, better 
resolutions is suggested by Berman (1980) in which he 
observes that some educational problems are amenable to 
programmz^tic (or explicit) solutions^ while others require 
mor^ complex, adaptive resolutions over time. Be that 
• as^it may, many innovative efforts have suffered from the 
lack of high quality, practical, usable resources. One 

of the more /fascinating' implications of this, observation 

1 

^ is to consider whether relatively Unstructured innovations 

> , ' - . .. I 

(e.g«., open educati'on) have..fjailed because they have 

eschewed'' the cumulative development of practical, specific 

resources at the operational; level. Bereiter and Kurland 

(1981) ' criticize the more unWtr,uctured educational 

proponents exactly on these grounds. . 

\ 
I 

I: 
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C haracteristics at the School District. Level * 

i, * 

The settings; in which people work - the planned and 

unplanned events and activities influence whether or not 

given change attempts will be productive. The local school 

system represents one major set x>f situational constraints f 

» 

or opoortunities for effective educational change. The^ same 
program is often successful in one school system, and on 

unmitiaated disaster in another. I have distilled a 

/ ' > 

substantial amount of evidence about what makes school systems 
effective into six factors: the history of innovative attenipts, 
the adoption process (if the change involves a district 
decision), central administrative sbpport and involvement, staff 
development approaches, tTTe~time-iine-alTd--±nf6rmation system, ' 
and board/community characteristics. 
5. The district's history of innovative attempts 

Most attempts at collective change in education seem » 
to fail, (with, some recent exceptions) and'fallure means 
frustration, wasted time, feelings of incompetence and 
lack of support, and disillusionment. The importance of 
the district's history of innovation attempts can be 
stated in the form of a proposition: .the moire that 
teachers or others have had negative experiences with 



■■■■ ' 

I do not consider every conceivable district factor. For example, 
the growing trend to more formal teacher/school board relationships 
(management/unioh contracts) could be examined, but the j-ssue^is too 
broad, -and insufficiently researched to warrant Sny generalizations 
about its impact on change. I have confined the list of factors to 
tho«e known to have an influence on'implementation. Factor 5 - the ' 
^ -district's history - of course, pertains to the political climate 
ERJC for change among the various groups. 
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previous implementation attempts in the district or 
elsewhere, the more cynical or apathetic they will be 
abont the next change which is presented regardless of the 
iperit of the new idea or program (see Sarason, 1971: 219--22) • 
Districts can develop an incapacity for change as well as 
a capacity for it (see Barman/ McLaughlin et al's 1978 
study of a distirict in a process of '*decay") . 
The adoption process 

The adoption procress (or decisions taken individually 
^or colleQtively to attempt certain changes or new dirjctions) 
in an important process in its own right (see especially 
.Crandall et al, forthcoming). Its relationship to 
implementation or change in practice is unclear. The Rand 
FPSEC research itidicated that opportuhistic\and * • 

bureaucratically- oriented adoption decisions are followed 
by limited impl-smentation (Berman and McLaughlin, 1978). 
As a result, of this adoption process, subordinates become 
indifferent to implementation, and senior management does 
not make serious , follow through attempts to provide" 
resources., training, etc. On the other -hand, if the 
decision to change has been carefully considered with 
appropriate commitment and follow through by the district, 
it is much more likeli to ;be taken seriously by principals 
or teachers. Berman, McLaughlin et.al's (1978) study of 
five schppl districts provide the clearest illustrations 
of how opportunities ana problem solving districts differ 
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in their day to day implementation of follow up 
activities (sec also Mi los\ .1^78) , five case studies},^ 

The ro3o of teachers in adoption decisions is 
particularly complicated as figure *two implies. 

Figure 2 Teacher Participation in Adoption 
Decisions and Scope of Change * 

Scope of Cha'nges 

♦ * 

Large Small , 



Teacher Partici- 
pation in Initial Yes 
Docii'ion 





^ 






i . 


II ■ 
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i 






III 


IV 
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For GxampVe^ even when teacher representatives are 
involved in program decisions (or development) dhout large 
r.cnlu change (Kiluatioii I), it docs noi' necessarily help 
other tojjchors when xt' comes to knowledge or cCmmitmcrL in 
implementing new ideas. Further, even when teachers arc 
not invt>lvcd in such initial decisions (situation III), > 
implementation may succeed if other factors are positive 
(e.%, the program addresses a need, staff development 
activities are established/ and so on,- see Crar.dal3 et al) 
In situations of small scale change, it is cjlear that with 
a small number^ of teachers involved it is preferable and 
feasible for most or all teachers to participate in major 
decisions. 



*Note., 1 am referring to initial adoption decisions. Teachers must 
be involved in implementation activities and decisions. 
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District ^dminrstrative support ' 

Individual teachers and single schools can bring 
about change without the support of central administrators 
but district or subdistrict wide change will not happen 
without -district support. Although *it has always 'been 
said that the superintendent (and the principal) are 
critical to educational change, it is only recently 
tl^t we arc beginning to understand more specifically 
what that means in practice- Most of the research cited 
in this section shows that the support of central 
administrators is critical for change in district practice 
(e.g., Emrick and Peterson, 19.78:76-73). It also shows 
that- general support or endorsement of a new program has 
very little inlrtiqnce on change in practice (for ej^ample, 
verbal support without implementation follow throug^) . 
Teachers and others know enough noW; if they didn^t 
fifteen years ago, not to take change seriously unless 
central administrators (demonstrate that, it should be,. 
(Borman, McLaughli.n et al, 1978:84-95 contains an 
excellent description of how one new superintendent with 
a mandate from the iDoard "transformed the organization i). 

. One of the more interesting analyses was carried out 
by Rosenblum and Louis' (1979:179). They investigated the 
relative efxects on implementation of superintendent 
authority on the 0^6 hand, and classroom autonomy , of the 
teacher on the other hand. Thfy found that superintendent 

. % 

authority (number of decision areas influenced by the 
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superintendent.) was positively associated with. 
ijnple;aentation, and- classroom autonomy (number of 
classrpom decisions 'that the teacher can make on his op her , 
own) was negatively related to implementation. 'Rosenbium 
and Louis suggest th^t degree of centraQJ^za^^ 
necess^ary for implemen€1^ng c^inpreliensive changes (across 
schools), and thatTstxong^ aorms of classroom autonomy 
in some districts may actually inhibit organizatipnal 
aiul di stri^'t 'wid(* riinntir*s- 

The basic point, however, is that the chief . executive 
offiror and other key central administrators set the 
conditions for implementation to the extent that they show 
specific forms- of '^support, and active knowledge and 
understanding of the realities of attempting to put a char.g- 
into practice. To state it most baldly, the. administrator ^ 
affects the quality of implementation to the extent that he 
or she understands and helps to manage the set of factors 
and the proces^^es described in this section. ^ 
Staff development and participation 

Sincfe the essence of educational change consists of 
learning neW ways of thinking and doing, new skills, 
knowledge, attitudes, etc., it follows that staff developmcr^t 
is one of the most important factors related to change in 
practice. Pre-implementation training by itself in which 
sessions, even intf.nsive ones,, are used to' orient people 
to new programs does not seem to work (Herman and McLaughlin, 
1.978:27, Downey et al,*1975. Miles, 1978, Smith "and Keith, 
197A, etc., etc.). One shot worlcshops prior to and even 
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duripg implementation are not very helpful (Rosenblum^ 
and Louis, 1979). Workshop trainters and program consultants 
are frequently ineffective. Consultants inside the district 
are unclear about their role and how to be -effective 
consultants (Simms, 1978, Lippitt, 1979). Teachers say 
they learn best from other teachers, but research shews 
that they interact wi-tli each other very infrequently 
(Lortie,, 1975). Whci) tfeachers are trained as staff 
dcve^^Opers they can be very effective in working with other 
tenchers"(sec Stnllings, 1981), Teachers also ,s.ny that ' 
they need., direct outside help, if it is practical and 
concrete which they find to be the exception rather than 
the rule. Researchers report that concrete and skill, 
specific trainiiig is effective, but "only for the short run" 
(McLaughlin and Morfeh, 1*978:76). . - 

The dilemmaa and inconsistencies in trying to understand 
why the "obvious" strategy of staff development fails more 
often than it succeeds are easy to resolve. Most staff 
de\:elopment activities consist of one-shot or oljher brief 
encounters with little or no follow through. When follow 
up and technical support occurs, there is strong evidence 
that it is related to implementation outcomes. Louis and 
Rosenblum (1981) for ekample, found that external assistance 
was very, important if it was characterized by intensity 
(frequent interactiorj) , initiation as well as responsiveness 
and involved a variety .of external agents with different 
skills. Research on implementation has demonstrated beyond 
a shadow of a doubt that processes of sustained interaction 
» ** 
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and staff development are crucial regardless of whether 
th'e change concerns federally funded change agent projects 

(Berman and McLaughlin, 1978), Louis and Rpsenblum, 1981. . 
Crandall et a'l, fcfrthcpming) , school district program 
change (Pratt and Melle, 1981, Berman, McLaughlin et ^1, 
1978), the National Diffusion Network (Emrick et al, 1977)-, 
Organization Development Progr^ams (Fullan, Miles anc Taylor 
1980), staff development programs related to implementation 

(Hatl and Loucks, 1978) , change in rural school districts 

(Herriott and Gross, 1979), Rosenblum and Loulfe,' 1979), 
•the planning and implementati9n of new schools (Kliles . 
et al, 1978), head stfert and follow through programs 

(Hodg<2S et'al>' 1980, Rhine (ed.) 1980, or technological 
'change (Yin et al, 1977). ^ , - •• ' 

Timerli ne and Information Systems 's- > 
_ ^ 

Time perspective is one of the most neglected asjiects , 
of the implementation process. Ten years ago, Sarasori.^ 
(1971) recognized time as a critical factor; 

"In -pradtice, the dfesire of the agents of change 'i:o'^ 

get started - not brtly because of internal and external 
. pressures but also because of the awareness, sometimes 

dim, that the road ahead will not be smooth - 'results' 
. in bypassing the different? aspects of the time, 

perspective problem, a bypass that might have no * . 

immediately adverse consequences, but can be counted 

on to produce delayed, and sometimes fatal difficulties 

(Sarason, 1971:219) 

Central decision makers know the" complexities of the 
adoption pyocess, practitioners know the complexities 
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of the implementation process. They live in two _ 
- 'different subjective worlds. What appears rational to 

one side, is resistance to change to the other (feee 
• Cdwden and Cohen, 1979).* Time is ignored,' because 
it cannot be -solved. There will never -be enough of it. 
Avoiding difficult problems creates even more serious 
ones, and time is no exception. .Unrealistic time lines ' 
add to the burdens of implementation ->materia'ls fail to . 
arrive on schedule, orientation and training is neglected 
or carried out perfunctorily, communication is hurried, 
•and frequently overlooked or misinterpreted, people 
become overloaded with the requiremen'ts qf new programs 
and carrying on as .usual (see , Charters and Pellegrin, 1973, 
Yin et al, 1977). Disillusionment burn out, "cynicism, 
. apathy, etc. come to characterize many people's orientation 
to a,ll changes tliat come along. . Open ended time-lines 
are also problematic because they create ambiguity about 
wh^ is expected and when, and a lack of . clarity about 
what constitutes progress. 

. . Whatever time-line is used, one of the major dilemmas 
faced i;5 what kind of information to collect, when, and 
how best to use it. This can range from highly elaborate 



*This helps to explain why each side feels misunderstood by the other, 
and why people are> frequently surprised by how others completely 
misinterpret the best of their intentions, and literally do not 
hear what they are saying. 
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accountability schemes (Wise, 1^980) to having no- formal; • ^ 
information system at all. Informati;on qah be ^ga-tfiered 
on the extent and progLems of impiemehtation:,- and/or on ^ _ 
'student achievement and other desired outcqmes. .The 
issues involved "in setting u^va district wide information 
system are too complicated to delve irito at this point 
exce'pt for three ob'servatigns . First there Is no evidence 
that information on student: achievement • by , itself results 
in, improved implementation. Student learning <3ata . «• ^ 
provides -little insight into the specific problems of 
implementation (such' as teaching,;^eh^vior , effective staff, 
development activities, etc.). Second, infqrjnation on. 
implementation concerns caa^be yery effec^iye i.n f aciiitating 
change. provided that it i$,/linked to a system for acting ^ 
on it. Pratt arid Melle/ l^'8i;. descritle how this works in 
one district. The research in general shows that most 
school districts do not have evaluation systems linked to 
instructional improvement procedures (Bank,. 1981-, l.yon 
et al, 1978).' Third, it is at the school and classroom- 
levels, where information counts. As I will indicate later 
an emphasis on- collecting and using information about 
student learning and other implementation problems has. 
been found to be strongly related" to school improvement ^ 
(Edmonds, 1979). . . . ) 
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10.^ CoiTununity Chara.cteristrcs 

It is very difficult to generalize about the role of ' 
communities in relation to implementation. Corwin (1973) 
fourid that community-school support Was correlated 
positiyeiy with innovatiyeness . Smith and Keith (1971) 
and Gold and Miles. (1981) describe, the painstaking sagas 

^ ' of what happen^ when mi^'dle class communities do not like 
the innovations they see in >^ their schools. DGnographic 
changes often put increasing pressure on schools to. adopt, 
if not implement, new policies (Herman and McLaughlin, 1979). 
For example, a case study of the Toronto school system' 
shows how the school board was central to the development 
of new multicultural policies and programs which were 

^ not neces^sarily welcomed by many schools (see Toronto 

School Board, 1976) • Rosenblum and. Louis (1979:111) found 
that "the degree to which environmental changes external 
to the school were impinging on it to ch&nge" was one of 
>four readiness factors related to subsequent implementation. 
Major conflicts, however, sometimes incapacitate districts 
. in bringing about 'actual change (in a sense, certain 

adoption decisions have to be settled before energy .can j 



be turned to implementation). VThatever the case, as Miles^ 
1980 asserts, attending to political stabilization in 
relation to the commur\ity is one of the primary tasks of 
planning and implementing' new programs. .In contemplating 
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or introducing innovatibhs, districts frequently' ignore 
the conununity (see Gold and ^iiles, 1981, Smith 'and Keith, 

1971) . ' * ' ^ • 

I would tentatively suggest: that most school . . ^ 

- ' * . • * , i 

communitdesva^re usually not directly involved in 

implementatioij; they can become aroused against certain 
innovations; neither highly stable 'or highly turbulent • 
school communities constitute effective environments for, 
'implementation^.* The role of individual parents in * 
■instruction (home tutors^ teacher aides) rather. than 
community groups may provide one of the most pov;erful 
leverages* to more effective implementation (Fantini, 1980), 
Armor et al, 19^6, and Fullan, forthcoming, chapter 12). 
School Level Factors ** . _ * ■ 

The meaning of the phrase "the school is the unit of «v • 
change" will become evident in this section. Three main factors 
summarize the influence of the school on implementation - the 
role of the principal, peer relationships, teacher orientations. 
Taken together they constitute the character and climate of 
"the school as an organization. 



- * 



There is no assumption implementation is "a good thing". /- 

Turbulent communities rieed to resolve, more basic pro^lcmg. regarding 
goals, p'urposes, and needed adoptions. • ^- 



I. do not/separXte elementary and secondary schools in this ^sectiqru 
There is f not enoWh rese^r.ph on "their differences in relation to 
effective change €o-a-llow for 'dear conclusions. The generalizations 
I do desdribe havd been found in schools at all levels (elementary, 
juhior high and secondary). 



ERIC 



•26 



11. ^ The Role o f the Principal 

— ' — z — — ^ — 

'< . - • 

Ali' ma jor,-^ research on innovation •shows that tljc 
principal strongdy influences the. likelihood of change, 
but It also indicates that most priiicipals do; not play 
' \ instructional leadership'' roles. Berman and McLaughlin ^ 
- ' (1977)', provide some detail 1 They found that "projects 

having the active suppoirt of the principal were the most 

■ » 

likely to fare '.veil" (p. 124, their emphasis). Principa'l^ ' 
actions^ serve to- legit^-mate whether a change is to' be 
taken seriously (and not all changes are) and to support 
, teachers (both psychologically and resource wise) . Berman 



and McLaughlin (1978:128) note that one of the best 
indicators of active involvement is whether the principal, 
attends workshop training sessions ♦ If we recall the 
earlier dimensions of change (beliefs, resourcj^s, teaching 
behavior), we might speculate that unless the ^principal 
gains some understanding' of these dimensions (not 
necessarily as an expert or. an instructional lea'der) he 
of she will not be able to ^understand teachers' concerns; 
that is,, will not be able to provide supporj: ^cr 
implementation. Such 'understanding re^quires interactio"n. 

* 

There is an abundance of other evidence which describes 
how 9nd why the principal is necessary for effective 
■ implementation. -To take . one example;, Emrick and 

Peterson's {1978.) synthesis of. five major projects^^uje 
, of which was the Rand project, FPSEC) identifies admm- 

istrative support as one of five factors common across 
^all projects: 
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"...utilization rarely occurred w,hen building 
ojt^ district administrative components were 
indifferent and utilization was virtually impossible' 
in the presence of- administrative opposition-^. 
(Emrick and Peterson, 1978:71) • ' - _ 

While 'the' principal can have a majo^ impact on 

implementation there is also considerable research v;hich' 

indicates that he or she frequently does- not play an 

active .role. Be'rman and McLaughlin (1978:131) report tjat 

one-tjird of the teachers thought that their principal 

functioned, priinarily as an adninistrator . Teachers rated 

these principals as ineffective and uninvolved in change. 



Similar f jndingfe come from studies of the use of provincial 
curriculum cjuidelines in Canada which indicate that at best 
6nly abolit oi^-\i^%t\ot school. principals provide active 
instructional leadership. (see Leithwood, Downey et al, 
1975, Siiwns, 1978). The subjective world of the principal 
is such that many of them suffer from the same problem in 
'••implementing a new role as facilitator of change!''as do 
teachers in implementing new teaching roles. What the 
principal. should do specifically to manage change at the 

school level is a complex affair, for which the "principal 

■""'1 ■'■>■'', ■ . ■ 

has little preparation. Th§ psychological and sociological 
problems of change Which confront the principal are at 
least as greaf as those experienced by teachers, (see 



"NT 



Leithwood and Montgomery, forthcoming and Fullan, 
1982/ Chapter : ' 
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12. Teacher-teacher relationshilps ' 

The theory of change which is evolving Trlearly ^ 

points to the importance of peer relat^^onships in the 

scfiobl. .Change involves-'^estfbialization. Interaction is 

the primary basis for sotial learning. New meanings, 

, new behavior, skills , ,etc*-, depend significantly on V 

whethter t'Gachers are working as isolated individuals 

'(Lortie, 1975, Sarason, .1971) or exchanging ideas, support, 

* and positive feelings about their work (Little,. 1981; ' 

Rutter et al., 1978). The researchll have, been reviewing^ ^ 

{Provides direct confiifTnatioh that the quality, of working 

^ ' relationships among teachers is strongly related to 

implementation (e.g., Berman and McLaughlin, 1978:119-120; 

Rosenblum and Louis, 1979; Miles et al, 1978). Collegiality , .. 

open communication, trust, support' and. help, interaction, 

' and morale are all closely related. How this comes about 

is another question,' but T have .already suggested that the 

principal strongly influences the climate of the school. 

Berman and' McLaughlin (1978), Galanter (1978)' ahd 'others 

report strong correlations between principal siipport and »i 

f 

peer relationships among teachers. 

13. Teachqf characteristics and orientaH:ions ^ ^ 

Research on teacher characteristics and effective 
change is inconsistent in its findings. "Level pf education > 
(e.g., possession of a master •s degree) and "years of 
teaiching experience are two variables frequently measured ^ 

29 - ■ , 
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in- reaearch studies. The results "vary, and it is not 
difficult to see why given the other factors in .table 1. 

It is not level .of education or 'years of 
experience that matter s'o much as. under what district 
and school conditions do teachers spend their time. ^' y 

Depending on the conditions, innovators and hard core 
resisters are found among all ages, and levels of <^ducationr: 
• , There is one teacher trait related to successful 
iinplernentation and studeat learning which comes through' 
strongly: teacher sense of efficacy > The Rand change 
agent study found a strong relationship between teacher 

ise of efficacy* and positive impact of change oh' 
varioiis^measures of success^ including percent of goals 
achieved, reports -of— impro^ed^^udent performance, and 
teacher change (Berman and McLau^hlirTT^T^^ . 

It» is more difficult to explain how teachers" ge^a 
' setise of efficacy, but it is encouraging to. observe that 
it is not idosyncratic. In some schools, there are much 
higher proportions of s.taff who possess . this, orientation 
than in others, even when community -and student 
. , , characteristics are similar. This suggests that efficacy 



*Defined as "a belief on the part of the teacher that he or she 
could help even thp most difficult or unmotivated students . 
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is more of an- prgahizatiortal feature of these schools 

which come to have a school-wide emphasis and expectation 

that they can improve student learning - and they do 

(Edmonds, 19.79; Rutter et al, 1979)- 

No words could sum up Section C more accurately than 

those of Judith Little,, based on her st^idy of work 

practices in six urban schools: 

"School improvement is most surely and thoroughly 
achieved when: 

Teachers enjgage in frequent, continuous and increasingly 
concrete and precise talk about teaphing practice (as 
distinct from teacher characteristics and failings, 
the social liyes of teachers, the foibles and failures 
of students and their families, and the unfortunate 
demands of society on the school) • By such talk, 
teachers build up a shared language adequate to the 
complexity of teaching,, capable of distinguishing one 
practice and its virtue from another... • 
Teachers and administrators frequently observe each' 
' other teaching, and provide each other with useful 
(if' potentially frightening) evaluatiphs of .their 
teaching. Only such observation and feedback can . 
# provide shared referents for the shared Janguage of 
teaching, and both demand and provide the precision . 
and concreteness which makes the talk about teaching 
useful. ' . 

Teachers and administrators plan, design, researc h. ^ 
evaluate and prepare teaching materials together . 
, The most prescient observations remain academic ("just 
theory") without the machinery to act on them. By 
joint work on materials', teachers and administrators 
share the considerable burden of development required 
by long-term improvement,. ^confirm their emerging under- 
standing of their approach, and make rising standards 
for their work attainalble by them and by their students. 
Teachers and administrators teach each other the 
practice of teaching (Little, 1981:1?-13^ her emphases). 

Only two of the six schools in Little's study evidenced 

a high percentage of these practices, but conditions for 
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plfectiveamplemen^tion on the part of individual 
teachers and administrators are clear. , 

External Eactors ' . ^, 

" - The I'ast set of factors are considered- under the general 
label of assistance external to the school .district ; 
Gbvernnients are by far the major direct and indirect sources of 
external assis-tSnce to school systems. Even with recent major 
cutbacks in federal expcdi turos- the combined federal and sta^ 
roles provide the margin required for impletotation support • 
in many school districts. As one might predict, whether it is - 
used fbr better implementation depends on the characteristics 

of local systems, that is, it depends on those factors just 
• described ia categories B and C. ^ Techni'cal. assistance for 
implementation (materials, consultancy , s,taff ..development , etc.) 
is frequently available in federal or state spbnsdred innovative 
programs. This too, involves a comialicated set of. issues. 
The amount of external assistance per se is unrelated to implement 
ation, but we are learning more about th& conditions under wMch 
external- help^is^eeded and effective, (see Louis and' Rosenblum, 
198I_; Louis, 1980, erahdall et. al forthcoming). The' simplest , 
observation at this juncture is that outside assistance or 
stimulation can influence 'implementation v^ry greatly provided 
that it is integrated with the factors at the<^local level 
described above. In particular, program a.ssistance (materials. 
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products) coinbined with certain types of technical assistance 
(external agents) is related to implementation (Louis and ^ 
Rosenblum) . — t * 

This completes an overview of factors affecting implementation. 
As Hall and Loucks (1977) observe, implemerttation is a process not 
an event. The factors described operate as a system of interacting 
variables having different values, combinations and contingencies 
in any given situation. It is extremely diffioiil'c to understand, 
let alone manage, the details of implementation in specific settings.*- 

Implementation Outcomes 

^ Attempts, at change have different kinds of outcomes.- I have 
alluded to the jnost obvious immediate one as "change in practice", 
or more specifically changes -in the use of new resources, new 
skills' and behaviors, and alteration of beliefs-. These changes 
of course, are presumably means to other outcomes such as increased 
student achievement. It xs only very- recently that we have begun 
to conceptualize and- attempt to measure implementation outcomes more 
comprehensively. The mbst helpful examination of the area of 
outcomes of change efforts is contained in the DESSI study (Crandall 
et al, forthcoming). Five/dif ferent kinds of outcomes are Identified 
and measured roughly in order from^ intermediate 'to more long term 
effects. 

1. Degree of Implementation 

2. Attitude toward Innovation 

ERIC ■ ■ 33 ■ 
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3» Impact (a) stjidents* benefits 
'(b) teachers' benefits 
ic) organizational benefits 

; -4. Continuation or Institutionalization * 
5* ' Attitude toward School Improvement 

...^ Degree of implementation assesses the degree of actual change 

on' the^ part oJE teachers (i.e,, it is similar to my definition of 

^'change in practice") . Attitude toward the innovation concerns 

perceptions of the strengths and weaknesses of the change • Impact 

invoi^^es an assessment of student learning, teacher benefits (e.g., 

professional development) J and organizational change (e.g., increased 

interaction^, teaming) Continuation involves such matters as 

incorporation in the budget., staffing and 'extent of durability of 

%he ch^noe./. Attitude toward school improvement is a kind of meta 

Tjari^ble related to whether the experience with the change effort 

incr,.eases dr decreases people's atti.tude toward engaging in new 

-> # . _ 

schdpl improvement programs - in brief, whether the experience has 

led people to conclude that it is worthwhile to try. and implement 
program changes. .- . 

The RDU study also makes a major contribution to the study of 
outcomes of adopted programs. Six different types of outcomes were 
identified: degree of organizational change, scope of implementation, 
incorporation of the product, incorporation of 4 problem-solving 
process i problem resolution, and personp.l impacts (Louis and 
Rosenblum, -1981:149) . 

Despite these important starts therfe are a^: 'least two major 

V 

problems which are intrinsically complicated. The first concerns 
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the difficulty of sorting out the. causal order among the different 
types of outcomes. Some are obvious such astchange in practice * 
hypothetically leads to increased student benefits.^ Others are- 
much more difficult to unravel such as the role of attitudes toward 
the change.' The second , and more fundamental problem is that the 
DESSI and RDU projects examined concrete innovations or products. 
Mahy other change initiatives are more open ended and difficult 
if not impossible to define in measurable terms (see Farrar et al, 
1980, Majone and Wildavsky, 1978, Fullan 1980). Thus, the 
assessment of outcomes is -much more problematic if one takes a more 
open ended (adaptive or evolutionary) view of change than if one 

_ 4 

examines innovations -from a fidelity or progra'mmatic perspective 
(see also Herman, 1980). " ' ' * 

• A final note of caution which, is implicit in the notion of 
the neejajto as sess a number of different types of outcomes, 
implementation per s^e is not necessarily a good thing. Change in 
practice may or may not lead to the desired outcomes. Certain 
outcomes may come at the" expense of others. The ^negative 
consequences ipay outweigh the positive on^s. The innovation or 
change may not be a- good one in relation to some people's goals, 
or in terms pf quality as was noted in'figtire one earlier. 
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III IMPLICATIONS 



We have progressed from general theories of failure toward 
more specific understanding of how educational change works. ^ 
Research during the past five years is increasingly more detailed 
and clear in, its explanations of both success and failure* Empirically, 
there are examoles of success {e;g., Crandall et al, forthcoming, 
Little, 1981, Louis and. Rosenblum, 1981, Stallings, 1981). They show 
that bifinging about change in practice is a careful .but nor over- 
whelming business. Small amounts of time and other resources used 
in a focused, persistent manner over a period of months can generate 
important benefits. Successful change processes, once described 
are easily understood because they are based on what seems to be 
••organized common sense." "(which is not to say that it is easy 'to 
harness comjnon sense in difficult situations.) ^ 

It is also encouraging to note that other bodies of research 

are independently confirming many of the findings ♦ Research on 

school effects (Edmonds, 1979, Rutter et al, 1979, D^Amico, 1980), 

on evaluation utilization (Bank, 1981), Kennedy et al, 1980, Lyon 

♦ 

et»al, 1978, and on irrservice education/staff develcpnent (Joyqe and Showers ,^ 1981) , 
Stallings, 1981, Little, 1981) report very similar results. While the convergence , 
is not perfect, the essential similarities serve .as a kind of external validation ^ 
that we are on to something worthvAiile. 

There are at least four broad issues requiring more work to 
which I will refer briefly here as (1) theories of contingencies, 
(21 theories of changing, (3) practical implications, and (4) question 
of the Impact of change. Relative tp the importance of contingency 
thinking, there are numerous viables. operating in any,*;given situation. 
ERsLC*^ described factor* of change in this at a middle range rather than - 
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a sltuationally specific range. ^ 

Questions of contingency rei'er to the combinations and interactive 
effects which occur in particular settings at particular times. 
Even lat the level of the 14 factors discussed in this paper th^re 
are numerous interactions and variations possible. When one moves 
to the level of specific eventk and to large numbers of social and 
personal variables affecting change processes and outcomes, the 
possibilities are countless. Yet we need theories and explanations 
closer to the action which do begin to map oat. contingencies and 
their conseguences (the 12 .case studies 'and press site analysis m 
the DESSI project is one impressive exajiple of what this entails 
and what benefits (more powerful, detailed explanations) accrue 
Crandall et al, forthcoming). 

Better comprehension of contingencies forms an important 

foundation for the second issue which should be addressed, namely, 
. ' ' * . 

the need for theories pf chan'ging — , how to move from A to B.* 
Case studies of successful ff forts', of course, do not contain 
direct lessions for how to obtain success in other situations. 

» 

The transfer of knowledge about change raises a whole new set oi 
issues. Some situations may not be amenable to any change attempts. 
In other situations, there are many different ways and places where 
one could start, and these will vSry. Is it better to put more 
emphasis on program development, leadership training staff 

> 

development, or climate change, or to do nothing, when one has to make ^ 



*To state the matter differently, "contingency" refers to the need 
for more detailed explanations , while "changing" refers to 
strategies for influencing situations in desired directions. 
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* choices about where to place resources. In su^, contingency 
thinking combined with -strategies for changing (whi^h variables in/ 
given situations are most ' important and most alterable) represents 

r * 

a critical line of future inquiry, if we are to build on the' 
considerable progress made in the past five year^. 
' ' The third issue — practical, implications — can be developed 
oniv to a certain extent in this paper. At one level the fourteen 
. factors can serve. as a checklist for approaching change more 
thoroughly. They offer in a direct way a set of guidelines for 
those aspects of change which should be planr^ed for, discussed 
and double checked throughout the process of change. Note also, 
,the message is that the fourteen factors must be viewed in 
combination, not in isolation from each other. . 

At an individual role level, however, we need more customized 
analysis and specific suggestions.' How to cppe with change will be 
a very different proposition if one is a teacher versus a principal 
vs. a parent vs. a district or state policy maker; and whether one 
is interested' in initiating change or forced to respond to programs 
initiated by others.* 

Th/ fourth issue is the big one. Does educational change, .pven 
. when implem'^n'ted make a difference^ in the short, intermediat6 and 

long run on student learning and life chances, on teacher satisfaction 
•and development, on the school ,as an organization? I/4ssume we will 
never have satisfactory answers to these -questions, bub the recent 



*In a recent book I have written separate'^chapters for each of the 
main roles including analyses of the sittfation in each role, and 
corresponding: guidelines for action. (Fullan, 1982). 
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work on conceptualizing and measuring a variety of outcomes 
should stimulate us to continue to struggle with these questions, 
and to move Jj^ward greater enlightment. In the. meantime, with 
only about twelve years of intensive research on problems of 
implementation behind us, we haven 't\done badly when you think 
about it. * 
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